Validity of the Agency for Healthcare Research and Quality clinical practice guidelines: how quickly do guidelines become outdated?

CONTEXT: Practice guidelines need to be up-to-date to be useful to clinicians. No published methods are available for assessing whether existing practice guidelines are still valid, nor does any empirical information exist regarding how often such assessments need to be made. OBJECTIVES: To assess the current validity of 17 clinical practice guidelines published by the US Agency for Healthcare Research and Quality (AHRQ) that are still in circulation, and to use this information to estimate how quickly guidelines become obsolete. DESIGN, SETTING, AND PARTICIPANTS: We developed criteria for defining when a guideline needs updating, mailed surveys to members of the original AHRQ guideline panels (n = 170; response rate, 71%), and searched the literature for evidence through March 2000 (n = 6994 titles yielding 173 articles plus 159 new guidelines on the same topics). MAIN OUTCOME MEASURES: Identification of new evidence calling for a major, minor, or no update of the 17 guidelines; survival analysis of the rate at which guidelines became outdated. RESULTS: For 7 guidelines, new evidence and expert judgment indicated that a major update is required; 6 were found to be in need of a minor update; 3 were judged as still valid; and for 1 guideline, we could reach no conclusion. Survival analysis indicated that about half the guidelines were outdated in 5.8 years (95% confidence interval [CI], 5.0-6.6 years). The point at which no more than 90% of the guidelines were still valid was 3.6 years (95% CI, 2.6-4.6 years). CONCLUSIONS: More than three quarters of the AHRQ guidelines need updating. As a general rule, guidelines should be reassessed for validity every 3 years.

The accuracy and effectiveness of routine population screening with mammography, prostate-specific antigen, and prenatal ultrasound: a review of published scientific evidence.

OBJECTIVE: To review published data regarding the accuracy and effectiveness of three screening tests: mammography, prostate-specific antigen (PSA), and prenatal ultrasound. METHODS: Published evidence regarding the accuracy and effectiveness of the three tests was collected by computerized literature search and supplemented by manual review of relevant bibliographies. RESULTS: Screening mammograms lower breast cancer mortality by about 20%. Most data come from women aged 50-64 years; women aged 40-49 years may also benefit, but the absolute risk reduction is lower. Up to 1,500 to 2,500 women must undergo screening to prevent one death from breast cancer. Mammograms miss approximately 12% to 37% of cancers, generate false-positive results, and cause anxiety while abnormal results are evaluated. PSA screening can detect 80% to 85% of prostate cancers but has a high false-positive rate. There is little direct evidence that early detection reduces morbidity or mortality. Indirect evidence includes a trend toward earlier stage tumors and steadily declining mortality rates in geographic areas where PSA screening has become common. Potential harms include the morbidity associated with evaluating abnormal results, and complications from treatment (e.g., impotence, incontinence). The overall balance of benefits and harms remains uncertain in the absence of better evidence. Prenatal ultrasound may reduce perinatal mortality, primarily through elective abortions for congenital anomalies, but does not appear to lower live birth rates. Although ultrasound has no proven effect on neonatal morbidity, it provides more accurate estimates of gestational age that prevent unnecessary inductions for post-term pregnancy. Screening detects multiple gestations, congenital anomalies, and intrauterine growth retardation, but direct health benefits from having this knowledge are unproved. Ultrasound has both positive and negative psychological effects on parents. The scans do not appear to harm childhood development. CONCLUSIONS: Even for the most established screening tests, the appropriateness of routine testing depends on subjective value judgments about the quality of supporting evidence and about the trade-offs between benefits and harms. Individuals, clinicians, policy makers, and governments must weigh the evidence in light of these values and the constraints imposed by available resources.

Current methods of the US Preventive Services Task Force: a review of the process.

The U.S. Preventive Services Task Force (USPSTF/Task Force) represents one of several efforts to take a more evidence-based approach to the development of clinical practice guidelines. As methods have matured for assembling and reviewing evidence and for translating evidence into guidelines, so too have the methods of the USPSTF. This paper summarizes the current methods of the third USPSTF, supported by the Agency for Healthcare Research and Quality (AHRQ) and two of the AHRQ Evidence-based Practice Centers (EPCs). The Task Force limits the topics it reviews to those conditions that cause a large burden of suffering to society and that also have available a potentially effective preventive service. It focuses its reviews on the questions and evidence most critical to making a recommendation. It uses analytic frameworks to specify the linkages and key questions connecting the preventive service with health outcomes. These linkages, together with explicit inclusion criteria, guide the literature searches for admissible evidence. Once assembled, admissible evidence is reviewed at three strata: (1) the individual study, (2) the body of evidence concerning a single linkage in the analytic framework, and (3) the body of evidence concerning the entire preventive service. For each stratum, the Task Force uses explicit criteria as general guidelines to assign one of three grades of evidence: good, fair, or poor. Good or fair quality evidence for the entire preventive service must include studies of sufficient design and quality to provide an unbroken chain of evidence-supported linkages, generalizable to the general primary care population, that connect the preventive service with health outcomes. Poor evidence contains a formidable break in the evidence chain such that the connection between the preventive service and health outcomes is uncertain. For services supported by overall good or fair evidence, the Task Force uses outcomes tables to help categorize the magnitude of benefits, harms, and net benefit from implementation of the preventive service into one of four categories: substantial, moderate, small, or zero/negative. The Task Force uses its assessment of the evidence and magnitude of net benefit to make a recommendation, coded as a letter: from A (strongly recommended) to D (recommend against). It gives an I recommendation in situations in which the evidence is insufficient to determine net benefit. The third Task Force and the EPCs will continue to examine a variety of methodologic issues and document work group progress in future communications.

Selection bias from requiring patients to give consent to examine data for health services research.

BACKGROUND: New rulings nationwide require health services researchers to obtain patient consent before examining personally identifiable data. A selection bias may result if consenting patients differ from those who do not give consent. OBJECTIVE: To compare patients who consent, refuse, and do not answer. DESIGN: Patients completing an in-office survey were asked for permission to be surveyed at home and for their records to be reviewed. Survey responses and practice billing data were used to compare patients by consent status. SETTING: Urban family practice center. PATIENTS: Of 2046 eligible patients, 1106 were randomly selected for the survey, were approached by staff, and agreed to participate. Approximately 87% of the nonparticipants were eliminated through a randomization process. MAIN OUTCOME MEASURE: Consent status. RESULTS: A total of 33% of patients did not give consent: 25% actively refused, and 8% did not answer. Consenting patients were older, included fewer women and African Americans, and reported poorer physical function than those who did not give consent (P<.05). Patients who did not answer the question were older, included more women and African Americans, and were less educated than those who answered (P<.02). Visits for certain reasons (eg, pelvic infections) were associated with lower consent rates. On multivariate analysis, older age, male sex, and lower functional status were significant predictors of consent. CONCLUSIONS: Patients who release personal information for health services research differ in important characteristics from those who do not. In this study, older patients and those in poorer health were more likely to grant consent. Quality and health services research restricted to patients who give consent may misrepresent outcomes for the general population. Arch Fam Med. 2000;9:1111-1118

Evaluating the administration of an office-based health survey in a primary care practice.

Self-administered waiting room questionnaires are popular tools for gathering health information from patients, but these data cannot be used for research purposes without confirming adequate sampling of the practice population and assessing the completeness and accuracy of patients' responses. Long-term data collection also requires avoiding an imposition on clinic operations. We developed a protocol to test these questions in a 9-week pilot study of 884 survey-eligible patients visiting a family practice clinic. We found an adequate proportion of eligible patients were approached (74%) and participated (89%), they provided relatively complete (82-98%) and accurate responses, and the impact on office operations was minimal (<2 min of staff time per participant). Some demographic differences in participation and survey item completion were identified. A systematic process for testing survey performance allowed us to not only document these findings, but also to rapidly identify problems and introduce solutions while the survey was in progress.

Health technology assessment in the United Kingdom.

The National Health Service (NHS) provides universal health coverage for all British citizens. Most services are free of charge, although modest copayments are sometimes applied. About 11% of the population also has private insurance. General practitioners, generally the first point of contact for accessing the system, are independent contractors who serve as gatekeepers for specialist and hospital services and enjoy substantial clinical autonomy. Hospitals are public and are regionalized, but the 1990 reforms made them self-governing trusts that contract with local purchasers (health authorities and general practitioner fundholders). Reforms beginning in 1990 moved the NHS away from a centralized administrative structure to more pluralistic arrangements in which competition, as well as management, influences how services develop. Health technology and health technology assessment (HTA) have gained increasing attention in the NHS during this period, as part of a wider NHS Research and Development (R&D) Strategy. The strategy promotes a knowledge-based health service with a strong research infrastructure and the capacity to critically review its own needs. HTA is the largest and most developed of the programs within the strategy. It has a formal system for setting assessment priorities involving widespread consultation within the NHS, and a National Co-ordinating Centre for Health Technology Assessment. The strategy supports related centers such as the U.K. Cochrane Centre and the NHS Centre for Reviews and Dissemination. A hallmark of the HTA program is strong public participation. The United Kingdom has made a major commitment to HTA and to seeking effective means of reviewing and disseminating evidence.

Evidence-based medicine. Interpreting studies and setting policy.

The ascendancy of EBM has been accompanied by a greater awareness of its shortcomings. It is increasingly evident from the cost, length, and difficulty of performing RCTs that studies cannot be launched to address every question in medicine. Good evidence is often lacking in medicine. Epistomologists question the very notions of evidence and the suitability of current study designs and measurement tools to research the salient issues of concern to patients and others concerned with quality. Lack of evidence of effectiveness does not prove ineffectiveness, yet, in reaction to EBM, insurance companies and government often make this inference to justify decisions to withhold coverage or research support. The unbridled enthusiasm for the evidence-based practice guideline of the early 1990s has been tempered by a more mature understanding of its limitations. Not many practice guidelines are developed well, and the implementation of flawed guidelines can cause harm. The seven-step process outlined earlier is slow, laborious, and expensive (sometimes costing hundreds of thousands of dollars). Moreover, there is little evidence that either the rigor of the methods or the guidelines themselves have a meaningful effect on practice behavior or patient outcomes. To the most cynical observers, the only consistent beneficiaries of guidelines are payers, who use guidelines with considerable success in reducing costs, lengths of stay, and utilization rates. Even ardent advocates of guidelines acknowledge the evidence that disseminating reviews and recommendations, by itself, fails to motivate clinicians to increase delivery of effective interventions and to abandon ineffective ones. This absence of response has stimulated a closer look at the barriers to behavior change and the design of thoughtful implementation strategies that begin with, but reach beyond, simple guidelines. Tools such as reminder systems, standing orders, academic detailing, peer review and audit, feedback, and health system changes recognize that knowing what to do is only one piece of an increasingly complex puzzle. The competitive marketplace of managed health care has added new economic influences on clinician behavior but is also fueling private-sector interest in good research. Patients, clinicians, and policy makers will continue to seek better data concerning what works in medicine and what does not.

A one-year audit of topics and domains in the Journal of the American Medical Association and the New England Journal of Medicine.

BACKGROUND: The relative emphasis that major medical journals give to topic areas has a potential effect on priorities in patient care, policy decisions, and public awareness. We measured the distribution of topics in two journals, by disease categories and domains, over a calendar year. METHODS: All original investigations, reviews, editorials, and special articles published in 1998 by the Journal of the American Medical Association and the New England Journal of Medicine were classified by article type, disease category, and domain. The 12 domains ranged from basic science to health policy, and included primary and secondary prevention. RESULTS: The 1159 articles published in 1998 included 889 (77%) articles about specific diseases-590 falling within eight specialties-and 190 (16%) articles on generic topics. Eighty (7%) articles concerned the behaviors that cause disease. Primary prevention and screening were the subject of 71 (6%) and 29 (3%) articles, respectively. Most of these concerned uncommon issues in patient care. Although 27 (2%) articles dealt with essential health promotion (e.g., diet, exercise), and none included a study on how to help patients to exercise, stop smoking, or eat a healthy diet. In contrast, 451 (39%) articles concerned the diagnosis and treatment of patients with disease. CONCLUSIONS: The relative emphasis that journals gave to prevention during the sample period seems discordant with its importance to patients and public health. Potential explanations include poor volume and quality of submitted research and editorial concerns about importance and reader appeal.

Controlling blood glucose levels in patients with type 2 diabetes mellitus. An evidence-based policy statement by the American Academy of Family Physicians and American Diabetes Association.

OBJECTIVE: To review evidence about the benefit of intensive glycemic control for patients with type 2 diabetes and to develop practice recommendations. PARTICIPANTS: A 9-member panel composed of family physicians, general internists, endocrinologists, and a practice guidelines methodologist was assembled by the American Academy of Family Physicians, the American Diabetes Association, and the American College of Physicians. EVIDENCE: Admissible evidence included published randomized controlled trials and observational studies regarding the effects of glycemic control on microvascular and macrovascular complications and on adverse effects. We followed systematic search and data abstraction procedures. Greater weight was given to clinical trials and to evidence about health outcomes. CONSENSUS PROCESS: Interpretations of evidence and approval of documents were finalized by unanimous vote, with recommendations linked to evidence and not expert opinion. The full report was prepared by the chair and 2 panel members, representing each of the 3 organizations. The initial draft underwent external review by 14 diabetologists and family physicians and changes consistent with the evidence were incorporated. CONCLUSIONS: The evidence demonstrates that the risk of microvascular and neuropathic complications is reduced by lowering glucose concentrations. Whether glycemic control affects macrovascular outcomes is less clear. The potential benefits of glycemic control must be balanced against factors that either preempt benefits (eg, limited life expectancy, comorbid disease) or increase risk (eg, severe hypoglycemia). The magnitude of benefit is a function of individual clinical variables (eg, baseline glycated hemoglobin level, presence of preexisting microvascular disease). Appropriate targets for treatment should be determined by considering these factors, patients' risk profiles, and personal preferences.

Developing an evidence-based Guide to Community Preventive Services--methods. The Task Force on Community Preventive Services.

Systematic reviews and evidence-based recommendations are increasingly important for decision making in health and medicine. Over the past 20 years, information on the science of synthesizing research results has exploded. However, some approaches to systematic reviews of the effectiveness of clinical preventive services and medical care may be less appropriate for evaluating population-based interventions. Furthermore, methods for linking evidence to recommendations are less well developed than methods for synthesizing evidence. The Guide to Community Preventive Services: Systematic Reviews and Evidence-Based Recommendations (the Guide) will evaluate and make recommendations on population-based and public health interventions. This paper provides an overview of the Guide's process to systematically review evidence and translate that evidence into recommendations. The Guide reviews evidence on effectiveness, the applicability of effectiveness data, (i.e., the extent to which available effectiveness data is thought to apply to additional populations and settings), the intervention's other effects (i.e., important side effects), economic impact, and barriers to implementation of interventions. The steps for obtaining and evaluating evidence into recommendations involve: (1) forming multidisciplinary chapter development teams, (2) developing a conceptual approach to organizing, grouping, selecting and evaluating the interventions in each chapter; (3) selecting interventions to be evaluated; (4) searching for and retrieving evidence; (5) assessing the quality of and summarizing the body of evidence of effectiveness; (6) translating the body of evidence of effectiveness into recommendations; (7) considering information on evidence other than effectiveness; and (8) identifying and summarizing research gaps. Systematic reviews of and evidence-based recommendations for population-health interventions are challenging and methods will continue to evolve. However, using an evidence-based approach to identify and recommend effective interventions directed at specific public health goals may reduce errors in how information is collected and interpreted, identify important gaps in current knowledge thus guiding further research, and enhance the Guide users' ability to assess whether recommendations are valid and prudent from their own perspectives. Over time, all of these advantages could help to increase agreement regarding appropriate community health strategies and help to increase their implementation.